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ABSTRACT 

Fast Patrol Boats were considered a negligible threat when the U.S. Navy focused 
on blue water operations away from shore. Now that the Navy’s focus has shifted to the 
littorals, where these ships patrol, the Fast Patrol Boat’s potential as a credible adversary is 
gaining acceptance. Moreover, the threat may be greatly enhanced if Fast Patrol Boats 
employ Commercial Off-The-Shelf Command and Control equipment to coordinate their 
efforts. This paper presents the design and results of a wargaming experiment conducted 
with Batman & Robin at the Naval Postgraduate School to examine this issue. 

The research question is how would U.S. Navy Surface Action Groups perform 
against Fast Patrol Boats equipped with Commercial Off-The-Shelf Command and Control 
equipment which enables these comparatively unsophisticated ships to conduct coordinated 
attacks. The experiment also looked at two other factors: Surface Action Group work load, 
which might exacerbate the offensive capability of Fast Patrol Boats conducting 
coordinated attacks with advanced Command and Control; and the quality of information 
our Surface Action Groups are provided which could likewise play a pivotal role in a 
confrontation or engagement. 

A two-cubed factorial experiment was conducted to test seven hypotheses. Data 
were collected on ten performance measures for 128 trials total. Significant results were 
obtained for three factors and three interactions. Operational explanations are provided. 
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EXECUTIVE SUMMARY 



Fast Patrol Boats (FPB) have always been an effective platform for littoral warfare. 
With today’s advanced commercial off-the-shelf command and control (C 2 ) technology, the 
threat FPBs pose to the U.S. Navy could drastically increase. Effective C 2 would enable 
FPBs to conduct coordinated attacks using sophisticated tactics that could saturate a Surface 
Action Group’s (SAG) defenses. 

In addition, the small size and high speed of FPBs combine to complicate 
intelligence gathering and increase the work load on SAG operators trying to maintain an 
accurate surface picture of the littoral environment. 

The main research question is the effect that coordinated attacks by FPBs, 
employing advanced C 2 , will have in an engagement with a U. S. Navy SAG operating in 
littoral seas. Also of interest are the effects, and their interactions with the main effect, of 
two elements which accompany the presence of FPBs and a littoral operating area: 
increased work load and incomplete information. 

FPB C 2 was modeled by varying the tactical coordination of the FPBs between high 
and low. High coordination was modeled by three concurrent waves of attacks of FPBs 
with most of the FPBs in a restricted emissions status. Low coordination was modeled by 
sequential attacks of FPBs. Work load was modeled by varying the number of neutral ships 
in the vicinity of the attack. High work load scenarios contained 24 neutral ships in the 
operating area while low work load scenarios contained only nine. Information 
completeness was modeled by varying the completeness of the maritime intelligence reports 
that the SAG received during the trial. High information completeness was represented by 
providing the subjects with a ratio of hostile ships in each threat sector to the total number 
of ships in the threat sector. Low information completeness was represented by only 
providing the total number of ships in each threat sector. 

A two-cubed factorial experiment was conducted to test the seven hypotheses (three 
main hypotheses plus four secondary hypotheses examining all possible interactions). Eight 
Persian Gulf scenarios were developed on the wargame simulator Batman & Robin as a 
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context for the experiment. The wargame has two modes of operation: Batman, where 
scenarios are executed, and Robin, where scenarios are developed. Each scenario 
represented a unique combination of the three factors at two levels. Sixteen subjects from 
the Naval Postgraduate School (NPS) participated in the experiment. Each subject played 
each scenario using Batman once. Trials (8 x 16 = 128) were run at four times real time and 
were completed in 38 minutes. 

In each scenario the Blue force SAG was comprised of three combatants and five 
helicopters. Red forces were comprised of nine FPBs. The subjects controlled the Blue 
order of battle. Red order of battle was scripted in Robin during scenario development but 
could deviate from the script in Batman according to its pre-programmed artificial 
intelligence. 

Ten performance measures were automatically generated by Batman & Robin at the 
end of each trial and were used to test the hypotheses. The data analysis plan includes use 
of frequency plots, box plots, MANOVA, univariate ANOVA, interaction plots, residual 
plots, and non-paramtetric statistics to assess the data. A significance level of a = 0.05 was 
established to test all null hypotheses. 

Hypotheses tests show significant results in the main hypotheses and several 
interaction hypotheses over a broad range of performance measures. The most important 
findings are condensed as follows: 

• When the FPBs had high C 2 , fewer were detected, however, of those detected 
the range of detection was greater. 

• When the FPBs had high C 2 , fewer were destroyed, but those that were 
destroyed were done so at a greater range. 

• The range at which the FPBs were detected and destroyed both decreased when 
work load increased. 

• With complete information the FPBs were destroyed at a greater range. 
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I. INTRODUCTION 



This chapter discusses the purpose and scope of the experiment and closes with the 
organization of the rest of the thesis. 

A. PURPOSE 

This section lays out the purpose of the experiment by describing the problem in 
real world terms. This leads to a statement of the research question, and the approach taken 
to examine the problem. The three main factors and their levels are discussed with 
anticipated results provided. 

1. Real World Problem 

As more and more navies turn away from blue water naval strategies that 
necessitate sophisticated and expensive multipurpose combatants and toward regional and 
littoral dominance, the fast patrol boat’s utility as a cost effective naval platform for littoral 
warfare increases [Ref. 1], Coupled with the availability of commercial off-the-shelf 
(COTS) command and control (C 2 ) technology, fast patrol boats present a unique challenge 
to the U.S. Navy and the traditional maritime powers still operating large naval combatants 
built for blue water operations [Ref. 2]. 

Moreover, the U.S. Navy frequently deploys Surface Action Groups or SAGs (task 
groups comprised primarily of combatants) throughout the littoral regions of the world. 
Despite the fact that SAGs have organic helicopter assets, without the protection of an 
aircraft carrier’s air wing they are vulnerable to stealthy, formidably armed fast patrol boats 
(FPB). 1 

The FPB’s small size and high speed combine to complicate intelligence gathering. 
In addition, U.S. Naval combatants are not optimized to track, target and neutralize small, 
fast surface platforms, nor can they still assume the dominant tactical advantage inherent 



1 The helicopter asset in mention is the LAMPS MK. III SH-60B Seahawk. Currently the Seahawk has no lethal 

ASUW capability. The SH-60B Block I upgrade will have a forward firing weapon’s capability. It is scheduled 
for fleet introduction by 1998 [Ref. 3]. 
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with superior C 2 [Ref. 4], Still employing forces specifically designed for blue water 
warfare, the U.S. Navy is caught in the middle of this shifting paradigm and appears slow to 
react to the FPB threat in the littorals [Ref 5]. 

2. Research Question 

FPBs already possess speed and stealth advantages over most combatants. The 
main research question concerns the effect that coordinated saturation attacks by FPBs, 
employing advanced C 2 , will have on an engagement with U. S. SAGs operating in littoral 
seas. Also of interest are the effects of increased workload and incomplete intelligence, 
both of which accompany the presence of FPBs in a littoral operating area, and their 
interactions with the main effect. 

3. Approach 

A three factor experiment was constructed involving human decision makers. All 
factors were presented at two levels, high and low, resulting in a two cubed factorial 
experiment. The experiment was designed and executed on the Battle Management 
Assessment System and Raid Originator Bogie Ingress wargame, more commonly called 
Batman and Robin (B&R). The wargame has two operational interfaces. In Robin , 
scenarios are developed and edited. In Batman , scenarios are executed and operational 
performance is measured. Eight different scenarios were generated in Robin to present all 
combinations of levels of factors. Each scenario was then administered to 16 Naval 
Postgraduate School (NPS) officer students using Batman, for a total of 128 trials. The 
geographic location of each scenario was identical: the Persian Gulf, between the Straits of 
Hormuz and Bahrain. Red forces were comprised of nine OSA II FPBs. A three combatant 
SAG composed of one Kidd class destroyer, two Spruance class destroyers, and five SH- 
60B Seahawk helicopters constituted Blue forces. All forces were armed and equipped 
realistically. The three design variables or factors were: 

a. Information Completeness 

For high information completeness, the maritime intelligence was 
represented by a ratio of hostile ships in a threat sector to the total number of ships in the 
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threat sector. For low information completeness, only the total number of ships per threat 
sector was given. 

b. Workload 

High workload scenarios were characterized by 24 neutral ships and nine 
hostile FPBs in the operating area. Low work load scenarios had nine neutral ships and 
nine hostile FPBs in the operating area. 

c. FPB C 2 

The third factor was FPB C 2 . This is not only the primary factor of interest, 
but also the original impetus for the experiment. The analysis will focus on this factor. 
Two levels of FPBs C 2 were achieved by varying their tactical coordination between high 
and low. High coordination was modeled by three concurrent waves of attacks of FPBs. 
For this case, seven of the nine FPBs were in restricted emission control (EMCON) status 
with the remaining two in an unrestricted EMCON status. It was assumed that coordinated 
attackers would share information about their enemy, the U.S. SAG in this case, and thus 
require fewer emissions. Low coordination was modeled by sequential attacks of FPBs 
with only two of the nine FPBs in a restricted EMCON status and the remaining seven 
emitting. 

4. Anticipated Results 

The hypothesized results were that friendly (Blue) forces would perform better 
when the following occurred: 

• Information or intelligence reports were complete. 

• The ship’s combat team was experiencing low workload conditions. 

• Incoming attacks were uncoordinated (sequential) vice coordinated (concurrent). 
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B. 



SCOPE OF THE EXPERIMENT 



The complexity of the scenarios for this experiment was limited due to time 
constraints and the varied backgrounds and limited operational experience of some of the 
subjects. 2 Blue Force resources were initialized in the same manner for every scenario and 
for every subject. This standardization was done to reduce the risk of subjects with 
operational experience gaining an advantage by relying on their expert knowledge to 
position their forces. In addition, certain artificialities were accepted in the interest of 
practicality and usability. For instance, the wargame was run at a speed of 4:1 (i.e., 4 
seconds of simulated time = 1 second of real time). Therefore a 152 minute simulated event 
could be compressed into 38 minutes of real time. All scenarios were constructed similarly 
except to allow for the systematic introduction of both levels of all factors. Scenarios were 
constructed to appear different to the subjects without affecting essential traits. This 
diminished the chance of subjects relying on- prior scenario knowledge to game the 
scenario instead of making decisions based on current scenario developments. 

C. ORGANIZATION 

The remainder of the thesis is organized as follows: Chapter II introduces the 
design of the experiment including the set-up, subjects, hypotheses, assumptions, statistical 
design of the experiment, and measures. Chapter III describes the data, how the raw data 
were collected and transformed, the data coding scheme, and data reduction phase. Chapter 
IV presents the data analysis plan, results of the data analysis and demographic information 
on the subjects. Chapter V presents conclusions including results of hypotheses tests and 
their real world meanings. Finally, Chapter VI offers recommendations to change, improve 
upon, and extend this experiment for the future. 



- The experiment had to be set-up, played, and recorded in the 50 minute class period. 
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II. EXPERIMENTAL DESIGN 



This chapter discusses experimental set-up and presents the seven hypotheses that 
are associated with this two-cubed experiment. In addition the assumptions, statistical 
design, and measures of the experiment are explained. 

A. SET - UP 

In this section the software, hardware, and subjects are discussed. B&R was chosen 
because of the ease and speed with which scenarios could be generated and modified. It 
addition, it has a superior graphical user interface and artificial intelligence which allows 
pre-scripted Red forces to deviate from the script in Batman and behave opportunistically. 
Finally, B&R has an automatic data collection capability which is very useful in the data 
collection phase. 

1. Software and Hardware 

B&R software was obtained from the creator of the wargame. Dr. Pat-Anthony 
Federico, at the Naval Personnel Research and Development Center (NPRDC), San Diego. 
It is written in the “C” programming language. B&R was installed in the NPS Systems 

Technology Lab (STL) on five Sun Sparc-20 stations and runs in a UNIX Solaris 2.5 

environment. All scenarios were created in Robin on one Sparc station and ported to the 
other four to ensure standardization. Each subject executed each scenario in Batman 
independently of other subjects. B&R is not capable of running in a multi-player 
networked configuration. 

2. Subjects 

Sixteen officers participated as subjects in the experiment. Before the experimental 
trials commenced, all were given a 50 minute standardization overview brief of: SAG 
tactics, platform capabilities, sensor and weapon performance parameters, weapon status 
and weapon posture meanings, and B&R rules. This was done for standardization purposes 
and in an effort to mitigate learning effects. Each subject was administered two practice 
trials followed by the eight experimental trials in which data was collected. During the 
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practice trials, subjects received one-on-one instruction on B&R from trained proctors. All 
trials were completed during a ten day period due to course requirement constraints at NPS. 
Subjects were limited to three trials per day to preclude fatigue. Subjects were not permitted 
to discuss their trials with other subjects before, during, or after trials. The experiment was 
conducted in a controlled environment with physical dividers placed on both sides of each 
subject’s computer station. Demographic information was collected on all of the subjects 
for future stratification analysis (see Chapter IV.B.4. for more information). 

B. HYPOTHESES 

Ten performance measures (PMs) or dependent variables were recorded for each 
trial. For a description of the PMs see Section G. The following hypotheses were tested for 
each PM: 

• H 01 : Information Completeness has no impact on SAG performance. 

• H a ,: Information Completeness impacts SAG performance. 

• H 02 : Work Load has impact on SAG performance. 

• H a2 : Work Load impacts SAG performance. 

• H 03 : FPB C 2 has no impact on SAG performance. 

• H a3 : FPB C 2 impacts SAG performance. 

• H n4 . The combined effect of Information Completeness and Work Load has no 
impact on SAG performance. 

• H a4 : The combined effect of Information Completeness and Work Load impacts 
SAG performance. 

• H 05 : The combined effect of Information Completeness and FPB C 2 has no 
impact on SAG performance. 

• H aS : The combined effect of Information Completeness and FPB C 2 impacts 
SAG performance. 
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• H 06 : The combined effect of Work Load and FPB C 2 has no impact on SAG 
performance. 

• H a6 : The combined effect of Work Load and FPB C 2 impacts SAG performance. 

• H 07 : The combined effect of Information Completeness, Work Load and FPB C 2 
has no impact on SAG performance. 

• H a7 : The combined effect of Information Completeness, Work Load and FPB C 2 
impacts SAG performance. 

The significance level for null hypotheses rejection was set at alpha = 0.05. For 
convenience, these hypotheses are repeated in Appendix I. 

C. ASSUMPTIONS 

Four assumptions were made at the outset of the trials: 

• Scenario outcomes represent the subject’s ability to act on or react to the cues 
displayed by Batman. 

• The ten performance measures (dependent variables) chosen from Batman 's 
automatic data collection capability adequately identified the impact or 
influence of the factors (independent variables) and their interactions on the 
outcome of each scenario. 

• The subjects’ performance will be normally distributed and all populations will 
have the same variance. 

• The scenarios are designed to measure the subjects’ performance over the 
dynamic range of the performance curve. 

D. STATISTICAL DESIGN OF THE EXPERIMENT 

The experiment was counter-balanced to negate any learning effect remaining after 
the practice trials. The two-cubed design of the experiment yielded eight unique scenarios. 
Using 16 subjects and a completely randomized block design with 2 replications yielded an 
8x16 output matrix. To construct the design matrix, the scenario type, order of scenario 
presentation and subject’s assignment to a scenario presentation were all randomized. This 
triple randomization was accomplished using one 8x8 Latin Square for rows 1-8; the 
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Latin Square was composed of the letters a-h. A duplicate Latin Square was then inverted 
and used to create rows 9-16. Using ‘APL2’ programming language, unique random 
numbers were randomly assigned to the eight columns and then ordered. The same process 
was carried out for the 1 6 rows. Since the initial scenario numbering scheme was a function 
of the original order of generation and included numbers greater than 8, the scenarios were 
renumbered arbitrarily from 1-8. Those numbers were then randomized and assigned the 
letters a-h. As a last step, we used a manual random number generator to randomly assign 
the subjects to the rows. B&R has a utility function enabling the administrator to assign 
subjects to classes, scenarios to tests, and finally, classes to tests. Assignments were based 
on the outcome of the counter-balancing process. This function minimized the chance of 
error inherent with administering 160 trials (32 training and 128 experimental trials). 

E. MEASURES 

B&R’s automatic data collection feature allowed 81 performance measures (PMs) 
or dependent variables to be collected after each trial. The 81 PMs are divided into two 
broad categories: Blue Forces’ performance vs. Red Forces, and Red Forces’ performance 
vs. Blue Forces. Each of these two broad categories is further divided into 3 sub-categories: 
air forces vs. surface, surface forces vs. surface, and combined air/surface forces vs. surface. 
Not all 81 PMs were applicable to this experiment. After a winnowing process the data was 
reduced to 10 PMs. Of the 10 PMs selected the first five measure Blue Forces’ 
performance and the second five measure Red Forces’ performance. The PMs used are 
defined as follows: 

• BAFP 1: FPBs detected by Blue Air (%). 

• BAFP 2: Average range (measured from nearest Blue force ship) at which 
FPBs were first detected by Blue Air (nm.). 

• BSP 3: FPB’s destroyed by Blue Surface (%). 

• BSP 4: Average range (measured from nearest Blue force ship) at which FPBs 
were destroyed by Blue Surface (nm.). 
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• BAFP 12: Number of TAO weapon assignments (count). 3 

• A 1 : Blue Air detected by the FPBs (%). 

• SI: Blue Surface detected by the FPBs (%). 

• A 2: Average range (measured from nearest FPB) at which Blue Air was first 
detected by the FPBs (nm.). 

• S 2: Average range (measured from nearest FPB) at which Blue Surface was 
first detected by the FPBs (nm.). 

• A 3: Blue Air destroyed by the FPBs (%). 



3 Blue forces had to utilize the TAO function in order to fire their weapons and engage Red forces. This number 
represents the number of salvos launched by the operator per trial. 
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III. DATA DESCRIPTION 



This chapter discusses the data collection and transformation process, the data 
coding scheme, and the data reduction phase. 

A. DATA COLLECTION AND TRANSFORMATION 

Although B&R calculated values for the PMs automatically, hardware limitations 
prohibited the automatic transfer of data to a file. Instead, each subject manually recorded 
their performance data displayed on the monitor to the PM data collection sheet at the end 
of each scenario. The large number of subjects participating at a given time precluded 
quality control while the subjects were recording their data. The information was then 
manually entered into Excel®. Before any data manipulations were performed, the data 
were put through a rigorous quality control process to check for data input errors. Data 
reduction was then performed and the data ported from Excel into Minitab® for statistical 
analysis. Once the data were in Minitab it was checked again for accuracy. 

B. EXAMPLES OF RAW DATA 

Appendix A contains an example of a subject’s PM data collection sheet (a PM 
sheet was generated after each trial). Notice that all 8 1 possible PMs were recorded on the 
PM data collection sheet. This was done so the data could be recorded quickly and 
accurately. As stated previously, only 10 PMs were actually used in the subsequent 
analysis. Appendix B contains a sample of the master data spreadsheet. The spreadsheet 
contains data for each of the 16 subjects organized by trial number, scenario number, and 
PMs results. Scenario numbers are described in the next section. 

C. DATA CODING SCHEME 

Each of the eight scenarios was uniquely described by the level, high or low, of each 
of the three factors: Information Completeness ( I ), Work Load ( L ), and FPB C 2 ( C ). 
This information was coded as a three digit number with each digit either a ‘2’ (high) or a 
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‘ 1 7 (low). Once the data from a PM sheet was entered into Excel, the scenario from which 
it was generated was referred to by this three digit code (see Appendix B, columns 3, 4, and 
5). For example, the scenario with I, L, and C at the high level is coded 222. The scenario 
with all factors at the low level is coded 111. Referencing sub-section I. A.4., we anticipated 
scenario 21 1 (high or complete information, low work load, low FPB tactical coordination) 
would produce the best results for Blue forces. 

D. DATA REDUCTION 

Initial data reduction was performed by reducing the 81 PMs which B&R was 
capable of producing for this experiment to the ten projected to be most relevant (see 
section II.E.). This was accomplished before the data collection phase. B&R automatically 
converts a subject’s performance to percentages, counts, and average distances and then 
lists them as performance measures. After the data collection phase the ten selected PMs 
were checked to ensure sufficient variability existed to warrant continuing with the analysis 
phase. Most of the PMs chosen produced good distributions. 
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IV. ANALYSIS 



This chapter discusses the overall analysis plan and presents the results of the 
analysis. The information in this chapter is quite detailed and necessary for a complete 
understanding of the results. A summary of findings is presented in the Conclusions 
(Chapter V). 

A. ANALYSIS PLAN 

Initial data analysis utilized frequency plots to obtain a pictorial representation of 
the 128 data points per PM (16 subjects x 8 scenarios). This established an initial feel for 
the data and provided insight for further detailed analysis. The shape of some of the plots 
may call into question our initial assumption of normality. Next, box plots were used to 
examine the three factors’ individual impacts on the PMs. After a working understanding 
of the data was obtained, three factor MANOVA was used as a screening device, and since 
not all PM distributions appeared normal, parametric and non-parametric ANOVA, 
interaction plots and residual plots were used to conduct a detailed analysis. A significance 
level of a = 0.05 was used to test all null hypotheses. When the data did not appear normal, 
non-parametric ANOVAs were conducted over the factors suggested by MANOVA, 
parametric ANOVA, or the box plots. 

B. RESULTS OF ANALYSIS 

Detailed results of the analysis presented in this section include frequency plots and 
the corresponding descriptive statistics, box plots, ANOVA, two-way interaction plots, and 
residual plots. In addition, the results of preliminary analysis performed on the 
demographic information is included. 

1. Frequency Plots and Descriptive Statistics (see Appendix C for plots) 

BAFP 1 (FPBs detected by Blue Air (%)): Almost all FPBs were detected by Blue 
Air throughout the experiment. This is not surprising; Blue Forces had five helicopters to 
conduct Surface Search Surveillance and Coordination (SSSC). 
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BAFP 2 (Average range at which FPBs were first detected by Blue Air (nm.)): The 
average range that Blue Air first detected the FPBs was 71 nm. measured from the nearest 
Blue force ship. 

BSP 3 (FPB’s destroyed by Blue Surface (%)): Overall 75% of the FPBs were 
destroyed. BAFP 1 and BSP 3 plots suggest that not all FPBs were destroyed even though 
almost all were detected. This is probably because Blue Forces expended their limited 
Harpoon inventory (8 weapons per ship) or were not able to position themselves within gun 
range due to the time constraints of the experiment. 

BSP 4 (Average range at which FPBs were destroyed by Blue Surface (nm.)): Blue 
Surface destroyed the FPBs at an average range of 30 nm. measured from the nearest Blue 
force ship (Harpoon range is 5-75 nm.; gun range is 1-15 nm.). 

BAFP 12 (Number of TAO weapon assignments (count)): The average number of 
TAO weapon assignments per game was approximately 19 (i.e., the number of salvos 
launched by the operator per trial). 

A 1 (Blue Air detected by the FPBs (%)): All Blue Air platforms were detected by 
the FPBs in most of the trials. Blue Air assets were required to come within the FPBs’ 
surveillance envelopes to positively identify contacts as hostile. 

S 1 (Blue Surface detected by the FPBs (%)): Overall 23.19% of Blue Surface 
platforms were detected by the FPBs. This is due in part to EMCON restrictions used by 
some FPBs. 

A 2 (Average range at which Blue Air was first detected by the FPBs (nm.)): The 
average range that the FPBs first detected Blue Air was 45.3 nm. Measured from the nearest 
FPB. 

S 2 (Average range at which Blue Surface was first detected by the FPBs (nm.)): 
The average range at that the FPBs detected Blue Surface was 17.4 nm. Measured from the 
nearest FPB. By combining plots SI and S2, it is apparent that the FPBs and Blue Surface 
platforms operated at distances greater than their surveillance areas. The initial separation 
of Blue Surface and the FPBs at the start of the game was great enough so that neither 
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would be in the other’s sensor range. This was done to force Blue to rely on the helicopters 
and intelligence provided to locate the FPBs. 

A 3: (Blue Air destroyed by the FPBs (%)): Overall 21% of Blue Air was destroyed 
by the FPBs. Blue Air was susceptible to the FPBs’ surface to air missiles while 
conducting visual identification. Electronic Surveillance Measures (ESM) information 
when available to Blue, may have made subjects more vigilant when operating air assets in 
the vicinity of unknown surface platforms. Without this information, the percentage of 
Blue Air destroyed could have been higher. 

2. Box Plots (see Appendix D for plots) 

As described earlier, the experiment examined the effects of three factors at two 
levels. Box plots were used to determine if there were significant differences between the 
distributions of the PMs for the two levels of treatment for each of the three factors. Box 
plots present a pictorial representation of the key features of the data: median, first and third 
quartiles, and spread. Significant differences in these features could indicate differences in 
the location or spread of the PM distribution when factor levels are altered. Within the 
context of these box plots, significant implies a non-trivial visual difference in the box plot 
with respect to the median and the first and third quartile. If differences were identified, 
ANOVA was conducted to gain additional insight. 

Four sets of box plots were generated for each PM. The first three correspond to 
individual factors. The boxes labeled with a 1 represent the 64 data points corresponding to 
the low level of the factor, whereas those labeled with a 2 represent data for the high level 
of each factor. The fourth set of box plot graphics contain eight box plots representing the 
eight unique scenarios (each scenario’s box plot contains 16 data points). It is coded in 
accordance with the data coding scheme described earlier. The box plots were used as a 
tool to help corroborate or refute the rejection of a null hypotheses and to gain insight into 
the nature of any effects. 
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3. ANOVA / Interaction and Residuals Plots (see Appendices E and F for 

plots) 

This sub-section builds on the results of the analysis based on the frequency plots 
and box plots and uses the additional statistical tools described earlier to make the case for 
acceptance or rejection of the null hypotheses (see Appendix I) for each PM. 

BAFP 1 (FPBs detected by Blue Air (%)): The Normal Plot of the ANOVA 
Residuals (NPR) suggests the data does not behave normally. This possibility is 
corroborated by the Histogram of the ANOVA Residuals Plot (HRP) and the ANOVA 
Residuals vs. ANOVA Fits Plot (RFP). ANOVA yields a p-value for C of 0.000. This 
implies that H 03 should be rejected, however, due to the non-normality of the data we can 
not reject H 03 based on ANOVA. The Kruskal-Wallis (K-W) non-parametric ANOVA, 
which does not require normality, was conducted and also yielded a p-value of 0.000. 
Based on the combined result, we reject H 03 . It makes sense that the level of C 2 would have 
an impact on Blue Air’s ability to detect the FPBs. The difference in means for Cl and C2 
is 98.09% detected and 90.10% detected respectively. Nothing else of significance was 
found. 

BAFP 2 (Average range at which FPBs were first detected by Blue Air (nm.)): The 
NPR, HRP and the RFP all indicate that the data is normal. ANOVA yields a p-value of 
0.001 and 0.000 for L and C respectively. The box plots show a median value for BAFP2 
that is approximately the same for LI and L2, however, the mean value of detection is more 
than 3 nm. greater when the workload is low (73.02 vice 69.63nm.). For these reasons we 
reject H 02 . H 03 is also rejected, however, the operational rational is not so intuitive. The 
boxplots show a significant difference in both mean and median with Blue Air detecting the 
FPBs further out at C2 probably because there was more time between engagements to 
move own forces out. The p-value for the LxC interaction is 0.059. Although this is very 
close to the significance level, the null hypothesis for H 04 will not be rejected. The 
Interaction Plots (IP) for LxC reveals little interaction. The p-value for IxLxC is 0.003. 
This leads us to reject 1 1 07 
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BSP 3 (FPB’s destroyed by Blue Surface (%)): The p-values based on ANOVA 
suggests not rejecting any null hypotheses except of H„ 3 (p-value =0.000). NPR, HRP, and 
the RFP all show the data to deviate from the normal. Hence, we will not base our decision 
on ANOVA but instead on K-W. K-W yields a p-value of 0.000 for H 03 ; on that evidence 
we reject H 03 . For H 03 , the box plots for Cl and C2 (medians = 88.89% and 66.67% 
respectively) indicate that fewer FPB’s were killed by Blue forces when under a 
coordinated attack (C2). This supports our primary expectation that Blue forces will 
perform worse when attacked in mass by FPBs using COTS C 2 . The IPs show no 
interactions exist for IxL, IxC, and LxC (p-values are 0.460, 0.460, 0.715, respectively) 

BSP 4: (Average range at which FPBs were destroyed by Blue Surface (nm.)): 
NPR, HPR, and the RFP all indicate normal behavior of the data. The p-value for I (0.047) 
indicates H 0 , should be rejected. The descriptive statistics indicate a significant difference 
between the means II and 12 (28.5nm and 31.78nm respectively). Therefore H 0I is rejected. 
We expect that with good information Blue forces will kill the enemy further out. For L 
and C the p-values also make the case to reject H 02/03 (0.001 and 0.002 respectively). The 
single factor box plot for L shows that Blue destroyed the FPBs further out when the load 
was light as we expect. The impact that factor C displays follows from BAFP2: since Blue 
Air detected the FPBs further out at C2, the FPBs destroyed were done so at a greater 
distance. This makes operational sense: if the enemy is detected further away then it can be 
destroyed at or near the outer limit of the weapon’s range. For all these reasons we reject 
H 02/03 . IxC yields a p-value of 0.011. The IP also suggests rejecting H 05 by showing a 
strong interaction between I and C therefore we reject H 05 . This synergy between I and C is 
most apparent when C is high and can be explained as follows: when the enemy’s attack 
plan is organized it is easier to predict, from present positional information, future 
movements based on extrapolation and to detect the enemy earlier (i.e., at a greater range) 
than when the attack is less coordinated (more random). IxL, LxC and IxLxC all have p- 
values »0.05. The IP for IxL and LxC confirms the acceptance of H 04/06 by showing little 
to no interaction. 
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BAFP 12 (Number of TAO weapon assignments (count)): NPR, HPR, and the RFP 
all indicate normal behavior of the data. The p-value for C is 0.000. The single factor box 
plot for C reveals a significant difference between the median number of TAO weapon 
assignments for Cl and C2 (23 and 18 count respectively). A likely reason for fewer 
weapon assignments being made when Blue was under a coordinated attack follows from 
BAFP1 : since fewer FPBs were detected when Blue was under a coordinated attack, fewer 
could be targeted. For these reasons we reject H 03 . No other p-values suggest rejecting a 
null hypothesis. 

A 1 (Blue Air detected by the FPBs (%)): NPR, HPR, and the RFP all indicate non- 
normal behavior of the data. ANOVA yields a p-value of 0.012 for factor C. K-W yields a 
p-value of 0.02. The single factor box plot for C shows that the FPBs detected far fewer 
helicopters when they were conducting a coordinated attack (this was expected since seven 
of the nine FPBs were in EMCON for C2). For these reasons H 03 is rejected. ANOVA for 
LxC (H 06 ) yields 0.032. The IP for LxC shows that more helicopters were detected by Red 
when the load for Blue was low (for both high or low C). The possible reason: when the 
load was low the helicopters were more accurately vectored to the FPBs. The IP for LxC 
also shows that when the load was high the percentage of helicopters detected by Red was 
about the same regardless of the level of C. For this we can find no clear operational 
explanation. 

S 1 : (Blue Surface detected by the FPBs (%)): NPR, HPR, and the RFP all indicate 
normal behavior of the data, however, the initial frequency plot of S 1 suggests the data 
may not be useful because there are only three Blue ships to detect. ANOVA yields a p- 
value of 0.003 for C. The single factor box plot for C registers a significant difference 
between Cl and C2 (medians = 33.0% and 17.0% respectively) confirming the low p- 
value’s indication that we reject H 03 (when C was high most of the FPBs were in EMCON). 
IxC p-value is 0.047. The IP also supports rejecting H 05 , however, the full operational 
explanation of the interaction displayed is unclear. When C is low the percentage of Blue 
Surface detected by the FPB’s increases when Blue has complete information (Blue closes 
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the distance with Red more promptly with complete information therefore Red also 
becomes aware of Blue’s presence). When C is high the opposite occurs. Perhaps this is 
because there are more unintentional encounters between Blue and the FPBs when 
information is incomplete. 

A 2: (Average range at which Blue Air was first detected by the FPBs (nm.)): 
NPR, HPR, and the RFP all indicate normal behavior of the data. ANOVA yields only one 
p-value below the cutoff: LxC = 0.000. The IP for LxC also suggests rejecting H 06 . The 
operational explanation of the nature of the interaction between LxC is unclear. 

S 2 (Average range at which Blue Surface was first detected by the FPBs (run.)): 
During 24 of the 128 trials Blue Surface was never detected by the FPBs (this occurred 
seven times at C 1 and 1 7 times at C2), unfortunately, that left the data set incomplete for 
this parameter. 4 The initial frequency plot and the box plots indicate that the data is not 
normal. The p-value for K-W with C as the factor was 0.081. The most we can say is that 
the mean value at which the FPBs detected Blue Surface is 17.42 nm. measured from the 
nearest FPB. The mean values for Cl and C2 are 17.7 and 17.08, respectively. 

A 3 (Blue Air destroyed by the FPBs (%)): The initial frequency plot indicates the 
data is skewed to the right. The NPR and RFP also indicate that the data is non-normal. 
ANOVA yields a p-value of 0.002 for C. K-W for C yields 0.053; not quite small enough 
to reject H 03 . The mean percentage of helicopters destroyed for Cl and C2 (15.4% and 
26.9% respectively) indicates that when the FPBs conducted a coordinated attack more 
helicopters were destroyed. This may follow from the fact that, although we could not 
reject H 03 for A2 (ANOVA p-value=0.164), the single factor box plot for C of that factor 
revealed a similar finding with the FPBs detecting Blue air further out when conducting a 
coordinated attack versus a non-coordinated attack (medians = 42 and 47nm. respectively). 



4 Analyzing the 24 times that Blue Surface was totally undetected using a chi-square test leads to rejecting the null 
hypothesis that Blue Surface is equally likely to go totally undetected for both Cl and C2 (p-value of 0.024). The 
estimated probabilities of Blue Surface going totally undetected are Cl : 0.1 1 and C2: 0.27. 
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It also makes sense that we will lose more scouts when the FPBs are in an EMCON status 
because Blue Air must enter the FPBs weapons envelope to positively identify them. The 
IP for LxC reveals when the FPBs were coordinated in their attack and Blue forces work 
load was high, more helicopters were destroyed. We already know C has an impact, but 
this indicates that Blue Air assets were not managed as carefully or prudently when the 
work load on Blue was high. 

4. Demographic Information 

Of the 16 subjects, eight were USN, five were USMC, two were USAF, and one 
was USA. Six of the eight naval officers were surface warfare officers. Thirteen of the 16 
subjects were students in the Command, Control, and Communications (C 3 ) curriculum at 
NPS. Years of service ranged from 5-16 years. A demographic questionnaire was 
completed by each subject during the overview brief; the questions were as follows: 

1. Name 

2. Rank / Branch of Service 

3. Designator / MOS Description 

4. Number of Years on Active Duty 

Subjects were asked to rate on a scale of from one (poor) to ten (expert) their: 

5. Level of Proficiency with Navy Tactical Data System (NTDS) symbology 

6. Level of Proficiency with Simulation / Wargames 

7. Level of Proficiency as Tactical Action Officer (TAO) 

8. Level of Computer Proficiency 

The answers to questions four through eight are discussed below: 

• The distribution of the years of service was skewed toward more senior officers 
(see Appendix G, Figure 1 for details). 

• The question on NTDS symbology was asked because the symbology used in 
B&R is very similar to NTDS symbology; it was presumed that familiarity with 
similar symbology would give a subject an advantage in adjusting to the 
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wargame. As expected, the distribution of the responses to this question were 
bimodal with ten of the sixteen subjects evenly split between the top end of the 
scale and the bottom end (see Appendix G, Figure 2 for details). 

• Experience with wargames was chosen early on as a factor that could influence 
a subject’s performance. The responses to the question regarding wargame 
experience appears to approach a normal distribution with responses skewed 
slightly toward more experience (see Appendix G, Figure 3 for details). 

• Another factor that was anticipated to impact performance was proficiency as a 
TAO. Since a TAO qualification is specific to SWO only, we assumed that the 
distribution would be skewed toward the more proficient side since the 
participant group contained a wide range of warfare specialties. A full fifty 
percent of respondents claimed the lowest level on the proficiency scale (see 
Appendix G, Figure 4 for details). 

• Finally, the data pertaining to the level of computer proficiency was evaluated. 
The data for this question appears to have a normal distribution skewed toward 
the less proficient end, with seventy five percent of the respondents claiming 
proficiency above the midpoint on the scale (see Appendix G, Figure 5 for 
details). 
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V. CONCLUSIONS 



This chapter discusses hypotheses results and provides a summary of the most 
significant results. In addition, corresponding real world interpretations are offered. The 
previous chapter was predominantly organized by PMs. This chapter is organized by the 
factors and interactions (hypotheses) and focuses on significant findings. Findings are also 
condensed and presented in table form in Appendix H (Null Hypotheses Rejection 
Summary). 

A. HYPOTHESES RESULTS 

All three factors and three out of four of their interactions produced significant 
effects, although not for all PMs. All significant effects are described below. 

1. Information Completeness 

H 0 , had a significant impact on one PM: The average range at which the FPBs were 
destroyed by Blue Surface (BSP 4); p = 0.047, 31.8 nm. when information was complete, 
and 28.5 nm. when information was incomplete. With complete information Blue forces 
were able to kill the enemy over 3 nm. further out than with incomplete information. 

2. Work Load 

H 02 had a significant impact on two PMs: The average range of detection (BAFP 2); 
p - 0.001, 69.6 nm. when Blue work load was high, and 73.0 nm. when work load was low. 
The average range at which the FPBs were destroyed (BSP 4); p = 0.001, 27.4 nm. when 
Blue work load was high, and 32.9 nm. when Blue work load was low. In both cases Blue’s 
performance was adversely affected (range decreases) when work load increased. 

3. FPB C 2 

H 03 had a significant impact on the following seven PMs: 

• Percentage of FPBs detected by Blue Air (BAFP 1); p = 0.000, 90. 1% detected 
when FPBs were coordinated, and 98.1% detected when the FPBs were not 
coordinated. 
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• Mean range of detection of the FPBs (BAFP 2); p = 0.000, 76 nm. when FPBs 
were coordinated, and 66.6 nm. when FPBs were not coordinated. 

• Percentage of FPBs destroyed by Blue Surface (BSP3); p = 0.000, 65.5% 
destroyed when the FPBs were coordinated, and 84.9% destroyed when the 
FPBs were not coordinated. 

• Mean range at which the FPBs were destroyed by Blue Surface (BSP 4); p = 
0.002, 32.8 nm. when the FPBs were coordinated, and 27.5 nm. when the FPBs 
were not coordinated. 

• Number of TAO weapon assignments (BAFP 12); p = 0.000, 17.0 assignments 
when the FPBs were coordinated, and 22.3 assignments when the FPBs were 
not coordinated. 

• Percentage of Blue Air detected by the FPBs (A 1); p = 0.012, 93.9% detected 
when the FPBs were coordinated, and 97.9% detected when the FPBs were not 
coordinated. 

• Percentage of Blue Surface detected by the FPBs (S 1); p = 0.003, 19.3% 
detected when the FPBs were coordinated, and 27.1% detected when the FPBs 
were not coordinated. 

When the FPBs used high C 2 fewer FPBs were detected by Blue, however, of those 
FPBs detected the range of detection by Blue was greater. Similarly, when the FPBs had 
high C 2 fewer were destroyed, but those that were destroyed were done so at a greater range. 
It only makes sense that if they were not detected they could not be killed, and if they were 
detected further out, that they would be killed further out. 

But why would fewer be detected and yet, those detected, be detected at greater 
range? The answer to this is inherent in the scenario design and is two fold. When the 
FPBs had high C 2 , they conducted a coordinated (concurrent) attack versus an 
uncoordinated (sequential) attack with seven of the nine FPBs in EMCON. Without the 
FPBs radiating they were more difficult to detect, but due to their more structured 
organization Blue had more time to prepare and deploy forces. 

Fewer TAO assignments were made when the FPBs used high C 2 . This explains 
why fewer FPBs were destroyed when C 2 was high. Analyzing the operations from the 
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FPBs perspective reveals that the FPBs detected fewer Blue Air and Surface platforms 
when they used high C 2 . This is a direct result and a potential disadvantage of the high C 2 , 
EMCON tactics. 

4. The Combined Effects of Information Completeness and FPB C 2 

H 05 had a significant impact on one PM: Mean range at which the FPBs were 
destroyed by Blue Surface (BSP 4); p = 0.011. When the FPBs were conducting non- 
coordinated attacks, information completeness had no effect. But when the FPBs used 
coordinated tactics, the FPBs were destroyed at a much greater range when Blue had 
complete information, 36.5 nm., than when Blue had incomplete information, 29.0 nm. 
This indicates that when Red was in EMCON, good intelligence (complete information) 
became more valuable. 

5. The Combined Effects of Work Load and FPB C 2 

H 06 had a significant impact on one PM even though neither factor taken alone was 
significant for that PM. 

Mean range at which Blue Air was first detected (A 2): Additional analysis does 
not yet help explain why the FPBs detected Blue Air further out when conducting 
coordinated attacks: p = 0.000, 46.14 nm. compared with 44.56 nm. when conducting non- 
coordinated attacks. 

6. The Combined Effects Of Information Completeness, Work Load, and 
FPB C 2 

H 07 had a significant impact on one PM. 

Average range of detection of the FPBs (BAFP 2): No attempt was made in this 
analysis to interpret this interaction. 

B. SUMMARY OF MOST SIGNIFICANT HYPOTHESES RESULTS 

• When Blue forces had complete information 

• FPBs were destroyed at a greater range. 

• When Blue forces were under low work load 
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FPBs were detected and destroyed at a greater range. 






• When FPBs attacked with high C 2 : 

• The number detected and destroyed decreased, and 

• The range of detection and destruction increased. 

C. REAL WORLD MEANING 

The fact that subjects had five helicopters scouting in a geographically constrained 
littoral region probably reduced the importance of judicious resource allocation and thus 
lessened the effect of information completeness. In essence, the subjects had nearly total 
radar coverage provided by the helicopters regardless of where they directed them. 

FPB C 2 levels caused changes in more PMs than any other factor. Probable causes 
include: 

• FPBs were in EMCON when C 2 was high which greatly affected their ability to 
see and be seen. 

• When FPB C 2 was high their tactics differed greatly than when their C 2 was low. 
In the former case, the FPBs converged on Blue in three concentric waves, 
thereby massing forces for saturation attacks. In the latter case, the attack was 
more sequential and random. The net result might be that even though Blue was 
able to conduct a more methodical and deliberate counterattack when the FPBs 
had high C 2 they could not detect as many and therefore could not kill as many. 
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VI. RECOMMENDATIONS 



This chapter proposes ways in which to improve and extend the experiment. 

A. CHANGES TO THE EXPERIMENT 

The experiment could be improved by more realistically modeling information 
completeness and work load. Reducing the number of helicopters would help determine if 
information completeness is modeled correctly. Fewer resources would force the operator 
to allocate them more judiciously. Work load could be varied between high and low based 
on real world maximum and minimum shipping levels in the scenario’s geographic area to 
ensure applicability to present day operations. The FPB threat is an emerging one so it will 
be more difficult to find real world data on which to model FPB C 2 and tactics but this 
aspect of the study should be pursued. 

Due to time and human resources constraints a limited pilot study was conducted 
(seven different subjects, three trials each). A larger pilot study would help ensure the 
factors are modeled more accurately and the PMs (dependent variables) are capturing the 
dynamic range of the performance curve. 

A more homogeneous group of subjects (preferably surface warfare officers) would 
help to further reduce the impact of the learning effect and ensure that all of the subjects 
were starting with the same baseline knowledge of naval tactics. 

Configuring the software to automatically store and transfer each subjects’ trial data 
would have expedited the data collection process and would have protected against 
transposition errors in the data. 

B. CONTINUATION OF THE EXPERIMENT 

The easiest way to extend this analysis is to segregate subjects by warfare specialty, 
service, or other demographic factor and then analyze the data accordingly. For instance, 
this would help determine if the changes in the levels of information were not registered in 
more PMs because of inaccurate modeling, or a cancellation effect due to certain subjects 
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utilizing the scripted message information prudently and others acting regardless of 
message content. Other ways include: replicating the experiment on another simulation 
system to show the affects of B&R artifacts, using different scenarios to neutralize scenario 
effects, or investigating different factors or different levels of studied factors. 
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APPENDIX A. PM DATA COLLECTION SHEET 
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1L Blue force air plritfd&ns detected ^ }’ >■. .j 


moa 


- \u.iA % 




2. Average range at which platforms were first detected 


45.3S nm 


a900 nm 


H\,S nm 


3. Blue force' air platforms destroyed 


TOGO % 


0,00 % 


^..Sb 3 % 


4. Average range at which platforms were destroyed 


t .50 nm 


Mali nm 


5 -eo nm 


5. Blue forcejalr mtelle detected > . -l .... '• 


d &u % 


- : ta>ao % 


t moo % 


6. Average range at which cruise missiles were detected 


ua ii nm 


K 0.23 nm 


■jtf- nm 


7. Blue force air cnji^jmisslles destroyed 


UoJJ % 


. 5.33 % 


9. S3 % 


8. Average range at which cruise missiles were destroyed 


kiaii nm 


0 00 nm 


0,00 nm 


9. Targets (HVO) destroyed fey surface platforms . 


o.nn % 


OCO % 


0,00 % 


1 0. Red force surface platforms destroyed 


0-w % 


49,49 % 


44.49 % 


1 1 . Average range at which rediplatforms destroyed 


UoJJ nm 


16. 35 nm 


IB.S5 nm 


1 2. Real time from beginning to end of scenario 


. 9 ft._ r ?,mln 


min 


PjB min 



Figure I. PM Data Collection Sheet 
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APPENDIX B. EXCEL WORKSHEET 



Name 


Trial 


I L C BAFP1 BAFP2 


BSE3 


BSP4 BAFP12 


A1 SI A2 S2 


A3 


Sub 1 


1 


1 2 


1 


100.00 


57.89 


66.67 


20.83 


26 


80 16.67 58.50 16.00 


0 




2 


2 1 


1 


100.00 


60.67 


66.67 


15.17 


30 


100 50.00 41.80 20.00 


0 




3 


1 1 


2 


100.00 


79.33 


55.56 


18.80 


16 


100 33.33 47.60 21.00 


0 




4 


1 2 


2 


77.78 


69.00 


44.44 


18.25 


10 


100 16.67 45.28 24.00 


80 




5 


1 1 


1 


100.00 


57 33 


77.78 


15.86 


20 


100 33.33 40.40 17.50 


0 




6 


2 2 


1 


88.89 


61.62 


66.67 


15.83 


21 


100 50.00 50.00 16.67 


0 




1 


2 1 


2 


100.00 


66.78 


44.44 


18.50 


18 


80 50.00 47.25 19.67 


0 




8 


2 2 


2 


100.00 


59.78 


66.67 


19.67 


19 


80 33.33 44.00 21.00 


0 


Sub 2 


1 


2 2 


2 


100.00 


80.56 


66.67 


38.17 


19 


100 0.00 46.00 N 


20 




2 


1 2 


2 


77.78 


79.57 


88.89 


25.75 


20 


100 33.33 38.40 17.00 100 




3 


2 1 


1 


100.00 


66.56 


88.89 


32.12 


26 


100 33.33 49.80 19.50 


20 




4 


2 1 


2 


100.00 


82.78 


88.89 


54.62 


19 


100 0.00 48.40 N 


40 




5 


2 2 


1 


100.00 


67.33 


88.89 


29.25 


32 


16.67 60.00 14.00 


20 




6 


1 2 


1 


100.00 


70.56 


100.00 32.00 


25 


100 33.33 40.00 19.50 


20 




7 


1 1 


2 


100.00 


89.56 


66.67 


25.67 


38 


100 50.00 49.60 16.33 


40 




8 


1 1 


1 


100.00 


69.22 


100.00 


34.89 


27 


100 16.67 31.00 14.00 


20 


Sub 3 


1 


2 2 


2 


88.89 


76.25 


77.78 


40.00 


12 


100 0.00 44.40 N 


0 




2 


2 1 


1 


100 00 


70.56 


88.89 


37.75 


13 


100 0.00 49.00 N 


0 




3 


1 2 


2 


66.67 


69.33 


55 56 


31.60 


12 


100 16.67 44.40 24.00 


60 




4 


1 1 


2 


100.00 


79.67 


66.67 


38.57 


18 


100 0.00 58.20 N 


20 




5 


2 2 


1 


100.00 


63.67 


88.89 


1962 


22 


100 33 33 49 00 20.00 


20 




6 


1 1 


1 


100.00 


62.00 


88.89 


25.62 


26 


100 33.33 38.60 19.50 


0 




7 


2 1 


2 


88.89 


76.62 


88.89 


31 12 


21 


100 33.33 42.40 20.00 


20 




8 


1 2 


1 


100.00 


60.00 


100.00 


19 44 


28 


100 33.33 49.00 17.50 


20 


Sub 4 


1 


1 1 


1 


100.00 


64 56 


88.89 


39 12 


16 


100 0.00 41.00 N 


0 




2 


1 2 


2 


77.78 


67.57 


33.33 


35.67 


7 


100 0 00 37.00 N 


80 




3 


2 1 


2 


88.89 


81.75 


77.78 


56.29 


9 


100 0 00 50.00 N 


0 




4 


2 1 


1 


100.00 


72.65 


88.89 


32.38 


15 


100 16.67 30.60 17.00 


20 




5 


1 2 


1 


100.00 


74.67 


77.78 


26.71 


21 


100 33.33 40.00 17.00 


20 




6 


2 2 


1 


100.00 


71.11 


88.89 


27 62 


22 


100 33.33 42.60 20.00 


20 




7 


1 1 


2 


100.00 


94.44 


77.78 


53.29 


10 


100 0.00 50.40 N 


40 




8 


2 2 


2 


100.00 


77.56 


77.78 


46.57 


9 


100 0.00 2640 N 


0 


Sub 5 


1 


2 2 


1 


100.00 


56.44 


77.78 


22.86 


16 


100 16.67 53.00 17.00 


20 




2 


2 1 


2 


100.00 


70.56 


77.78 


37.00 


23 


100 0.00 39.00 N 


0 




3 


1 1 


2 


100.00 


68.33 


66.67 


21.67 


17 


80 16.67 45.00 18.00 


0 




4 


1 2 


2 


88.89 


63.00 


55.56 


15.60 


22 


100 33.30 40.20 17.00 


80 




5 


2 2 


2 


100.00 


78.11 


55.56 


28.68 


16 


100 16.67 41.00 14.00 


40 




6 


1 2 


1 


88.89 


63.62 


77.78 


28.57 


12 


100 16.67 39.40 14.00 


20 




7 


2 1 


1 


77.78 


68.43 


77 78 


28.57 


13 


100 16.67 39 80 14.00 


60 




8 


1 1 


1 


100.00 


71.78 


100.00 46.11 


17 


100 0.00 40.00 N 


20 



Figure 2. Worksheet, Subjects 1-5 
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Name 


Trial 


I L C BAFP1 BAFP2 


BSP3 


BSP4 BAFP12 


A1 SI A2 S2 


A3 


Sub 6 


1 


2 


2 


1 


100.00 


64.89 


77.78 


26.86 


35 


80 33.33 48 75 16.50 


20 




2 


1 


1 


2 


44.44 


78.75 


22.22 


18.50 


25 


80 16.67 48.25 14.00 


0 




3 


2 


1 


2 


44.44 


75.75 


22.22 


30.00 


28 


60 33.33 58.67 15.50 


0 




4 


2 


1 


1 


88.89 


71.62 


77.78 


26.29 


31 


100 16.67 49.60 15.00 


0 




5 


2 


2 


2 


55.56 


74 80 


22.72 


19.00 


22 


100 16 67 49.60 14.00 


0 




6 


1 


1 


1 


88.89 


65 38 


77 78 


26.29 


26 


100 16.67 42.40 14.00 


20 




7 


1 


2 


2 


77.78 


79.14 


33.33 


24.67 


25 


80 16.67 54.75 15.00 


0 




8 


1 


2 


1 


100.00 


69.11 


77.78 


22.43 


29 


100 16.67 48.80 15.00 


0 


Sub 7 


1 


1 


1 


2 


88.89 


76.00 


77.78 


17.57 


18 


100 33.33 45.00 17.50 


40 




2 


2 


2 


1 


100.00 


69.56 


88.89 


23.38 


15 


100 33.33 41 60 15.00 


20 




3 


1 


2 


1 


100.00 


67.11 


88.89 


21.50 


19 


100 50.00 47.60 18.33 


20 




4 


1 


1 


1 


100.00 


67 89 


88.89 


17.62 


30 


100 33.33 41.20 14.50 


0 




5 


1 


2 


2 


88.89 


73.38 


55.56 


12.60 


18 


100 33.33 30.80 19.00 


20 




6 


2 


1 


1 


100.00 


69.44 


88.89 


22.28 


29 


100 33.33 39.60 17 00 


0 




7 


2 


2 


2 


100.00 


78.22 


77.78 


25.00 


18 


80 16.67 44.25 19 00 


40 




8 


2 


1 


2 


100.00 


81.67 


77.78 


42.00 


17 


60 16.67 59.67 14.00 


0 


Sub 8 


1 


2 


1 


2 


77.78 


67.14 


33.33 


56.33 


20 


60 0.00 40.67 N 


0 




2 


2 


2 


1 


100.00 


64.33 


77.78 


13.29 


34 


80 50.00 41.25 16.33 


0 




3 


1 


1 


1 


100.00 


61.89 


88.89 


24.38 


27 


80 33.33 56.50 17.50 


0 




4 


1 


2 


1 


100.00 


57.00 


77.78 


17.14 


43 


80 33.33 57.50 21.00 


0 




5 


2 


1 


1 


100.00 


68.89 


100.00 26.44 


24 


100 33.33 39.80 20.50 


20 




6 


1 


2 


2 


100.00 


72 67 


66.67 


28.50 


19 


100 33.33 45.60 14.00 


40 




7 


2 


2 


2 


100.00 


67.78 


88.89 


24.88 


19 


100 16.67 46.80 14.00 


20 




8 


1 


1 


2 


100.00 


75.67 


77.78 


34.57 


11 


80 16.67 56.50 14.00 


20 


Sub 9 


1 


1 


2 


2 


44.44 


70.75 


33.33 


22.67 


10 


80 16.67 46.75 20.00 


60 




2 


1 


1 


1 


100.00 


66.44 


88.89 


39.38 


13 


100 16.67 41.40 20.00 


40 




3 


2 


2 


2 


100.00 


74 56 


66.67 


47.33 


13 


100 0.00 42.20 N 


40 




4 


2 


2 


1 


100.00 


74.11 


88.89 


42.62 


14 


100 0.00 46.80 N 


40 




5 


1 


1 


2 


100.00 


90.78 


66.67 


42.00 


17 


100 16.67 50.00 25.00 


20 




6 


2 


1 


1 


100 00 


70 33 


100.00 


34.00 


16 


100 50.00 30.40 20 00 


20 




7 


1 


2 


1 


100.00 


72.67 


100.00 


38.89 


11 


80 0.00 47.75 N 


40 




8 


2 


1 


2 


100.00 


82.67 


77.78 


45.86 


6 


100 16.67 47.60 14.00 


20 


Sub 10 


1 


1 


1 


2 


77.78 


75.71 


44.44 


32.25 


21 


100 0.00 46.80 N 


20 




2 


1 


2 


1 


100.00 


68.78 


77.78 


42.00 


0 


100 16.67 56.00 15.00 


0 




3 


2 


2 


1 


100.00 


67.56 


88.89 


24.62 


27 


100 50.00 55.60 19.00 


20 




4 


2 


2 


2 


100.00 


81 11 


77.78 


48 14 


10 


80 16.67 42.75 15.00 


40 




5 


1 


2 


2 


100.00 


67.44 


66.67 


25.17 


23 


80 33.33 32.25 17.50 


80 




6 


2 


1 


2 


100.00 


82.44 


100.00 


50.22 


18 


80 16.67 45.00 14.00 


0 




7 


1 


1 


1 


100.00 


71.67 


88.89 


34.88 


27 


100 33.33 33.60 20.00 


20 




8 


2 


1 


1 


100.00 


71.78 


100.00 41.33 


12 


100 0.00 39.60 N 


0 



Figure 3. Worksheet, Subjects 6-10 
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Name 


Trial 


I L C BAFP1 BAFP2 


BSP3 


BSP4 BAFP12 


A1 SI A2 S2 


A3 


Sub 11 


1 


2 1 


1 


100.00 


71.67 


66.67 


26.67 


27 


100 16.67 60.00 14.00 


20 




2 


2 2 


2 


100.00 


77.56 


44.44 


27.50 


19 


100 16.67 46.00 18.00 


0 




3 


1 2 


1 


100.00 


70.78 


77.78 


28.00 


23 


100 33.33 42.20 16.00 


0 




4 


1 1 


1 


100.00 


69 00 


100.00 23.67 


23 


100 16.67 40.60 20.00 


20 




5 


2 1 


2 


88.89 


83.00 


77.78 


33.00 


13 


100 16.67 55.30 14.00 


0 




6 


1 1 


2 


88.89 


80.75 


88.89 


30.38 


11 


100 0.00 57.20 N 


20 




7 


2 2 


1 


100.00 


65.00 


88.89 


22.50 


25 


100 16.67 34.40 20.00 


20 




8 


1 2 


2 


100.00 


75.56 


66.67 


24.17 


23 


100 33.33 47.00 14.50 


80 


Sub 12 


1 


2 1 


1 


100.00 


61.11 


77.78 


25.57 


29 


100 33.33 45.40 17.50 


80 




2 


1 2 


1 


100.00 


73.00 


77.78 


24.29 


21 


100 33.33 47.40 17.00 


40 




3 


2 2 


2 


100.00 


82.56 


66.67 


39.00 


16 


100 33.33 50.00 15.50 


60 




4 


2 2 


1 


100.00 


62.67 


88.89 


30.00 


23 


100 16.67 50.20 15.00 


40 




5 


2 1 


2 


100.00 


81.67 


66.67 


44.33 


23 


100 16.67 48.40 19.00 


20 




6 


1 2 


2 


88.89 


71.25 


66.67 


29.83 


23 


100 16.67 40.60 15.00 


80 




7 


1 1 


1 


100 00 


68.33 


88.89 


22.00 


36 


100 50.00 40.00 20.00 


20 




8 


1 1 


2 


100.00 


69 78 


100.00 32.67 


21 


100 33.33 48.20 14.00 


0 


Sub 13 


1 


1 2 


2 


88.89 


71.00 


66.67 


23.83 


17 


100 33.33 4740 17.00 


80 




2 


2 2 


2 


100.00 


78.22 


66.67 


44.50 


15 


100 0.00 53.28 N 


20 




3 


1 1 


1 


100.00 


68.22 


88.89 


32.25 


18 


100 16.67 33.60 20.00 


0 




4 


1 2 


1 


100.00 


69.56 


88.89 


19.62 


22 


100 50.00 40.20 18.33 


40 




5 


1 1 


2 


100.00 


89.67 


77.78 


42.14 


10 


100 16.67 43.00 14.00 


40 




6 


2 1 


2 


100 00 


77 44 


77 78 


31.14 


17 


100 33.33 47 00 17.00 


0 




7 


2 2 


1 


100.00 


56.25 


88.89 


20.25 


20 


100 50.00 42.20 20.33 


60 




8 


2 1 


1 


100.00 


71.11 


88.89 


21.75 


25 


100 33.33 40.40 17.58 


0 


Sub 14 


1 


1 1 


1 


100.00 


64.44 


77.78 


22.00 


26 


100 33.33 39.60 19.50 


0 




2 


2 1 


2 


100.00 


75.78 


77.78 


39.43 


14 


100 16.67 48.20 14.00 


0 




3 


1 2 


2 


77.78 


72.00 


77.78 


25.43 


15 


100 33.33 39.80 17.00 


80 




4 


1 1 


2 


100.00 


77 78 


88.89 


30.88 


19 


100 16.67 49.40 15.00 


20 




5 


1 2 


1 


100.00 


67.11 


88.89 


22.88 


23 


100 33.33 38.60 17.00 


0 




6 


2 2 


2 


88.89 


76.62 


77.78 


37.00 


12 


100 0.00 46.60 N 


40 




7 


2 1 


1 


100.00 


71.11 


100 00 41.44 


11 


100 16.67 49.40 19.00 


20 




8 


2 2 


1 


88.89 


60.75 


88.89 


20.62 


28 


100 50 00 41.00 20.00 


40 


Sub 15 


1 


1 2 


1 


100.00 


58.33 


55.56 


26.20 


8 


80 33.33 46.25 16.50 


0 




2 


1 1 


2 


66.67 


70.67 


11.11 


32.00 


1 


80 0.00 50.00 N 


0 




3 


2 1 


1 


100.00 


64 56 


77.78 


22.71 


22 


100 50 00 43.80 16.33 


0 




4 


2 1 


2 


100.00 


70.78 


66.67 


19.83 


18 


100 50.00 56.00 20.00 


0 




5 


1 1 


1 


100.00 


61.33 


66.67 


26.50 


14 


100 16.67 44.80 18.00 


20 




6 


2 2 


2 


100.00 


75.11 


55.56 


15.60 


18 


100 33.33 48.20 16.50 


0 




7 


1 2 


2 


88.89 


71.25 


55.56 


17.40 


15 


100 50.00 45.80 18.33 


40 




8 


2 2 


1 


88.89 


61 62 


77.78 


21.14 


20 


100 33.33 39.40 14.50 


0 



Figure 4. Worksheet, Subjects 11-15 
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Name 


Trial 


I L C BAFP1 BAFP2 


BSP3 


BSP4 BAFP12 A1 


SI 


A2 


S2 


A3 


Sub 16 


1 


2 


1 


2 


44.44 


72.75 


44.44 


39.00 


21 


80 


0.00 


35.50 


N 


20 




2 


1 


1 


1 


88.89 


67 75 


77 78 


41.00 


21 


100 


000 


47 40 


N 


0 




3 


2 


2 


1 


77.78 


69.43 


66.67 


25.67 


27 


100 


33.33 


55.20 


19.50 


20 




4 


2 


2 


2 


88.89 


64.00 


77.78 


38.29 


18 


100 


16.67 


39.00 


19.00 


20 




5 


2 


1 


1 


100.00 


69.44 


100.00 


45.44 


19 


100 


16.67 


48.00 


19.00 


0 




6 


1 


1 


2 


100.00 


89.44 


100.00 


52.67 


21 


100 


16.67 


47.40 


15.00 


0 




7 


1 


2 


1 


100.00 


71.78 


88.89 


34.12 


23 


100 


16.67 


50.20 


20.00 


20 




8 


1 


2 


2 


100.00 


72.56 


66.67 


43.00 


17 


80 


50.00 


46.25 


19.67 


20 



Figure 5. Worksheet, Subject 16 
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APPENDIX C. FREQUENCY PLOTS 




Figure 7. Frequency Plot for BAFP 2 
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Frequency Frequency 




Figure 9. Frequency Plot for BSP 4 
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Frequency Frequency 




Figure 10. Frequency Plot for BAFP 12 




Figure 11. Frequency Plot for A1 
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Frequency Frequency 
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Figure 12. Frequency Plot for SI 




Figure 13. Frequency Plot for A2 
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Frequency Frequency 
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Figure 14. Frequency Plot for S2 
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Figure 15. Frequency Plot for A3 
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APPENDIX D. BOX PLOTS 




Figure 16. BAFP 1 Box Plots 
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Figure 17. BAFP 2 Box Plots 
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Figure 18. BSP 3 Box Plots 
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Figure 19. BSP 4 Box Plots 
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Figure 20. BAFP 12 Box Plots 
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Figure 21. A 1 Box Plots 
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Figure 22. S 1 Box Plots 
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Figure 23. A 2 Box Plots 
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Figure 24. S 2 Box Plots 
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Figure 25. A3 Box Plots 
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APPENDIX E. INTERACTION PLOTS 




Figure 26. BAFP 1 Interaction Plots 
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Interaction Plot - IVfeans for BAFP 2 




Figure 27. BAFP 2 Interaction Plots 
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Interaction Plot - IVfeans for BSP 3 




Figure 28. BSP 3 Interaction Plots 




Figure 29. BSP 4 Interaction Plots 
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Interaction Plot - Maans for BAFP 12 




Figure 30. BAFP 12 Interaction Plots 




Figure 31. A 1 Interaction Plots 
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Figure 32. S 1 Interaction Plots 
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Figure 33. A 2 Interaction Plots 




Figure 34. S 2 Interaction Plots 
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Figure 35. A3 Interaction Plots 
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APPENDIX F. RESIDUALS PLOTS 



Rssidual IVbdd D agnostics BAFP1 



Normal Rot of Rsa'duals 



I Chart of fea duals 
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Figure 36. BAFP 1 Residuals Plots 
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Residual Model D agnostics BAFP2 
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Figure 37. BAFP 2 Residuals Plots 
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Figure 38. BSP 3 Residuals Plots 
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Figure 39. BSP 4 Residuals Plots 
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Figure 40. BAFP 12 Residuals Plots 
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Figure 41. A 1 Residuals Plots 
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Figure 42. S 1 Residuals Plots 
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Figure 43. A 2 Residuals Plots 
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Figure 44 . A3 Residuals Plots 
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Figure 45. Q4: Years Active Duty 
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Figure 46. Q5: Proficiency with NTDS Symbology 
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Figure 47. Q6: Proficiency with Wargames 
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Figure 48. Q7: Proficiency as TAO 
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Q8: Computer Proficiency 
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Figure 49. Q8: Computer Proficiency 
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APPENDIX H. NULL HYPOTHESES REJECTION SUMMARY 





BAFP1 


BAFP2 


BSP3 


BSP4 


BAFP12 


A1 


SI 


A2 


S2 


A3 










♦ 














h„ 2 




♦ 




♦ 














H„3 
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♦ 
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♦ 








h„ 4 






















if) 
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X 








♦ 














Hoe 
















♦ 






h„ 7 




♦ 



















Note: Reading this table horizontally provides a summary of the PMs for which the given null hypotheses were rejected. Reading the 
table vertically provides an indication of the number of null hypotheses rejected for the given PM. 



Table 1. Rejection Summary 
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APPENDIX I. NULL HYPOTHESES 



H 01 


Information Completeness has no impact on SAG performance. 


H al 


Information Completeness impacts SAG performance. 


h 02 


Work Load has no impact on SAG performance. 


H a2 


Work Load impacts SAG performance. 


H„3 


FPB C 2 has no impact on SAG performance. 


H a3 


FPB C 2 impacts on SAG performance. 


h 04 


The combined effect of Information Completeness and Work Load has no impact on 
SAG performance. 


H a4 


The combined effect of Information Completeness and Work Load impacts SAG 
performance. 


h 05 


The combined effect of Information Completeness and FPB C 2 has no impact on 
SAG performance. 


H a5 


The combined effect of Information Completeness and FPB C 2 impacts on SAG 
performance. 


h„ 6 


The combined effect of Work Load and FPB C 2 has no impact on SAG performance. 


H a6 


The combined effect of Work Load and FPB C 2 impacts on SAG performance. 


h 07 


The combined effect of Information Completeness, Work Load and FPB C 2 has no 
impact on SAG performance. 


« a7 


The combined effect of Information Completeness, Work Load and FPB C 2 impacts 
SAG performance. 



Table 2. Null Hypotheses 
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